Up to now, a technique of performing variable processing on an input picture to generate a picture of higher picture quality has been developed. By utilizing this technique, it is possible to generate, from a noisy input picture, a picture reduced in noise, or to generate a picture having its edge not blurred, from an input picture containing a picture area having relative movement with respect to other picture portions, such as background, referred to below as a feature area, e.g., a telop, and having the edge blurred.
Referring to FIGS. 1 to 9, the conventional method of generating a picture reduced in noise is explained.
In a conventional picture processing apparatus 10, a noisy input picture is input to a substracter 11 and to an amplifier 14. The substracter 11 is also fed from a frame memory 17 with a picture delayed one frame with respect to the input picture. The picture from the frame memory 17 is also supplied to an amplifier 16.
The substracter 11 subtracts the picture, sent from the frame memory 17, from the input picture, to send the resulting difference to a difference calculation unit 12. The difference calculation unit 12 acquires the absolute value of the results of the calculations from the substracter 11, as a difference between the input picture and the picture from the frame memory 17 (picture of the directly previous frame picture) to output the so acquired absolute value to a threshold value decision unit 13.
The threshold value decision unit 13 compares the difference value from the difference calculation unit 12 to a preset threshold value and, based on the results of comparison, decides whether or not the input picture is a still picture portion or a moving picture portion, from pixel to pixel, to set a weighting value p based on the result of the decision.
If the input picture is determined to be the still picture portion, the weighting value p is set to a preset fixed value between 0 and 0.5. If the input picture is determined to be the moving picture portion, the weighting value p is set to a value 1.
The amplifier 14 amplifies the signal of the input picture, with the weighting value p, as set by the threshold value decision unit 13, as an amplification factor, to send the amplified signal to an adder 15. The amplifier 16 amplifies the signals of the picture from the frame memory 17, using a value corresponding to subtraction of the weighting value p from unity (1), as an amplification factor, to send the resulting amplified signal to the adder 15, which then sums the outputs of the amplifiers 14, 16 together to output the resulting sum signal.
That is, if the input picture is the still picture portion, the pixel value of the input picture and the pixel value of a picture delayed by one frame from the input picture are weight-summed together, with a weighting value p. By summing the noisy input picture to the picture of the directly previous frame, in this manner, temporally non-steady elements, such as noise, may be reduced.
However, if the input picture is a still picture portion, the weighting value p is set at a certain fixed value, and hence the noise cannot be reduced in association with its magnitude or direction. Moreover, if, due to mistaken decision by the threshold value decision unit 13, the moving picture portion is determined to be the still picture portion, weight-addition with the weighting value p is also applied to the moving picture portion, so that, in this case, trailing type picture quality deterioration is produced.
If the input picture is the moving picture portion, the weight-addition with the weighting value p=1 is executed. That is, the input picture is output directly, such that the noise is not reduced for the moving picture portion. On the other hand, if the input picture is the still picture portion, but the noise contained therein is significant, the still picture portion may be erroneously determined to be the moving picture portion. In such case, the input picture (still picture portion) is directly output, that is, the noise is not reduced.
FIG. 2 shows another illustrative structure of a conventional picture processing apparatus. In a picture processing apparatus 20, shown in FIG. 2, the noise of the moving picture portion is reduced by classification adaptive processing which is based on taps corresponding to the motion vector.
The input picture, corrupted with noise, is sent to a frame memory 21-1. The frame memory 21-1 delays the input picture by one frame and sends the so delayed input picture to a frame memory 21-2, a motion vector detection unit 22 and to area extraction units 24, 27.
The frame memory 21-2 delays the picture from the frame-memory 21-2 by one frame and sends the so delayed picture to a frame memory 21-3, a motion vector detection unit 23 and to the area extraction units 24, 27.
The frame memory 21-3 delays the picture from the frame-memory 21-2 by one frame and sends the so delayed picture to the motion vector detection unit 23 and to the area extraction units 24, 27.
The motion vector detection unit 22 applies motion matching to two temporally consecutive frames, sent from the frame memories 21-1, 21-2, to detect the motion vector to send the detected results to the area extraction unit 24.
The motion vector detection unit 23 applies motion matching to two temporally consecutive frames, sent from the frame memories 21-2, 21-3, to detect the motion vector to send the detected results to the area extraction unit 27.
The area extraction unit 24 references the motion vector supplied from the motion vector detection unit 22 to extract preset picture areas as class taps from the frames supplied from the frame memories 21-1 to 21-3.
FIG. 3A shows picture areas extracted as class taps. A sum-total of three pixels, namely a subject pixel on a frame Fn from the frame memory 21-2, as indicated with a black circle in the drawing, a pixel on a frame Fn−1 from the frame memory 21-1 (frame directly previous to the frame Fn), lying at a position specified by the motion vector from the motion vector detection unit 22 with respect to the subject pixel, as indicated by a hatched circle, and a pixel on a frame Fn+1 from the frame memory 21-3 (frame directly following the frame Fn), lying at a position specified by the motion vector from the motion vector detection unit 22 with respect to the subject pixel, as indicated by a hatched circle, are extracted as being class taps.
If, for example, the frame Fn and the other frames are as indicated in FIG. 3B, that is if the motion vector between the frame Fn−1 and the frame Fn is (−1, −1) and the motion vector between the frame Fn+1 and the frame Fn is (1, 1), with the motion vector between the frame Fn and the frame Fn being naturally (0, 0), as shown in FIG. 3B, the subject pixel on the frame Fn, the pixel on the frame Fn−1 at a position specified by the motion vector (−1, −1) with respect to the subject pixel and the pixel on the frame Fn+1 at a position specified by the motion vector (1, 1) with respect to the subject pixel, are extracted as being class taps, as shown in FIG. 3C.
The area extraction unit 24 sends the extracted class taps to a class code generating unit 25.
The class code generating unit 25 applies e.g., ADRC processing to the class taps supplied from the area extraction unit 24 and extracts tempo-spatial patterns of the class taps while generating a class code indicating the class obtained on classification conforming to the extracted patterns. The class code generating unit 25 sends the generated class code to a ROM table 26.
The ROM table 26 holds a set of prediction coefficients calculated for a case where a noisy picture is a pupil picture and a noise-free picture is a teacher picture, from class to class, and outputs what corresponds to the class code sent from the class code generating unit 25, from among the prediction coefficients of the set, to an estimation processing unit 28.
The area extraction unit 27 references the motion vector supplied from the motion vector detection unit 23 to extract preset picture areas as prediction taps from picture data of the frames supplied from the frame memories 21-1 to 21-3.
FIG. 4A shows the structure of the prediction taps. A subject pixel on the frame Fn, indicated with a black circle in the drawing, along with pixels lying around the subject pixel, indicated with thin black in the drawing, totaling at 13 pixels, a pixel on the frame Fn−1 specified by the motion vector from the motion vector detection unit 22 with respect to the subject pixel, along with pixels lying around the pixel, indicated with thin black in the drawing, totaling at 13 pixels, and a pixel on the frame Fn+1 specified by the motion vector from the motion vector detection unit 22 with respect to the subject pixel, along with pixels lying around the pixel, indicated with thin black in the drawing, totaling at 13 pixels, are extracted as being prediction taps.
If, for example, the motion vector between the frame Fn−1 and the frame Fn is (−1, −1) and the motion vector between the frame Fn and the frame Fn+1 is (1, 1), with the motion vector between the frame Fn and the frame Fn being naturally (0, 0), as shown in FIG. 4B, the subject pixel on the frame Fn, a pixel on the frame Fn−1 at a position specified by the motion vector (−1, −1) with respect to the subject pixel and a pixel on the frame Fn+1 at a position specified by the motion vector (1, 1) with respect to the subject pixel, along with each 13 pixels there around, are extracted as being class taps, as shown in FIG. 4C.
The area extraction unit 27 sends the extracted prediction taps to the estimation processing unit 28.
The estimation processing unit 28 executes preset calculations, based on the prediction taps supplied from the area extraction unit 27 and on the set of the prediction coefficients, supplied from the ROM memory 26, to generate a picture reduced in noise.
However, in the present instance, since the block matching is first carried out to detect the motion vector, a large amount of calculations are needed, thus entailing extremely time-consuming operations.
FIG. 5 shows another illustrative structure of the conventional picture processing apparatus. In a picture processing apparatus 30, shown in FIG. 5, classification adaptive processing based on taps corresponding to the motion vector is similarly carried out to reduce the noise in the moving picture portion. However, in the present case, the motion vector is detected by a method in which the processing load is lesser than with the block matching.
This picture processing apparatus 30 is provided with a tap array decision unit 31, which is to take the place of the motion vector detections units 22, 23 of the picture processing apparatus 20 of FIG. 2. It is noted that parts or components equivalent to those shown in FIG. 2 are denoted by the same base numerals to omit the corresponding description for simplicity.
A noisy input picture is sent to a frame memory 21-1. This frame memory 21-1 delays the input picture by one frame and routes the so delayed input picture to a frame memory 21-2, an area extraction unit 24, an area extraction unit 27 and to a tap array decision unit 31.
The frame memory 21-2 delays the picture from the frame memory 21-1 by one frame and sends the so delayed picture to a frame memory 21-3, area extraction units 24, 27 and to the tap array decision unit 31.
The frame memory 21-3 delays the picture from the frame memory 21-2 by one frame and sends the so delayed picture to a frame memory 21-4, area extraction units 24, 27 and to the tap array decision unit 31.
The frame memory 21-4 delays the picture from the frame memory 21-3 by one frame and sends the so delayed picture to a frame memory 21-5, area extraction units 24, 27 and to the tap array decision unit 31.
The frame memory 21-5 delays the picture from the frame memory 21-4 by one frame and routes the so delayed picture to the area extraction units 24, 27 and to the tap array decision unit 31.
The tap array decision unit 31 detects the motion vector from the frame memories 21-1 to 21-5, and decides the arraying positions of the class taps or prediction taps, based on the so detected motion vector, to send the results of decision to the area extraction units 24, 27.
The motion vector is detected in the tap array decision unit 31 based on the following premises:
In five consecutive frames, the interval between the first frame (frame number 1) and the last frame (frame number five) is short, with the picture movement over the five frames being a rectilinear movement at an equal speed.
If the noise is contained in none of the five consecutive frames, the variance of pixels of the respective frames lying at the same position of the picture is equal to 0 or extremely close to 0.
That is, in these premises, a straight line segment may be drawn which passes through a subject pixel on the frame Fn (frame from the frame memory 21-3), and pixels lying in register with the subject pixel, that is lying at the same positions on the picture, on the frame Fn−1 (frame from the frame memory 21-2), frame Fn−2 (frame from the frame memory 21-1), frame Fn+1 (frame from the frame memory 21-4) and frame Fn+2 (frame from the frame memory 21-5), in a three-dimensional plane comprised of X axis and the Y axis on the picture and time, as shown in FIG. 6. That is, the tap array decision unit 31 detects this straight line segment as being a motion vector.
Referring to the flowchart of FIG. 7, the operation of the tap array decision unit 31, in case of executing the processing of tap array decision, is hereinafter explained.
At step S1, the tap array decision unit 31 sets, on the frame Fn, an area comprised of, for example, 5×5 pixels, centered about a subject pixel on the frame Fn from the frame memory 21-3.
At the next step S2, the tap array decision unit 31 selects a pixel from the area of the 5×5 pixels on the frame Fn, as set at step S1, as the center pixel, while setting a search range centered about a pixel on the frame Fn−1 from the frame memory 21-2, registering with the center pixel.
At step S3, the tap array decision unit 31 initializes the coordinates (a, b), specifying a pixel in the search range, to (0, 0), as shown in FIG. 8.
At the next step S4, the tap array decision unit 31 generates a straight line segment passing through the pixel in the search range, specified by the coordinates (a, b), that is a pixel on the frame Fn−1, and through the subject pixel on the frame Fn.
At step S5, the tap array decision unit 31 calculates the variance between the pixels of the frame Fn−2 to Fn+2, lying on the line segment generated at step S4. At step S6, the tap array decision unit 31 verifies whether or not the so calculated variance is smaller than the variance values held at step S7, as later explained, that is whether or not the variance calculated at step S6 is the smallest of the variance values calculated for straight line segments passing through the pixels of the coordinates (a, b) checked so far and through the subject pixel. If the variance calculated is found to be of the least value, the program moves to step S7. At this step S7, the tap array decision unit 31 holds the variance value calculated at step S5 and the associated coordinate (a, b).
If the variance calculated is found at step S6 not to be of the least value, or if the variance value and the coordinates (a, b) are held at step S7, the program moves to step S8, where the tap array decision unit 31 updates the coordinates (a, b). The updating of the coordinates (a, b) may be done on a raster scan.
At the next step S9, the tap array decision unit 31 verifies whether or not the coordinates (a, b) updated at step S8 exceed the search range as set at step S2 (FIG. 8). If it is verified that the search range is not exceeded, the program reverts to step S4 to execute the subsequent processing. If conversely the search range is found to be exceeded, the tap array decision unit 31 assumes that the search for the search range has been finished and proceeds to step S10 to vote for the coordinates (a, b) held at step S7.
At the next step S11, the tap array decision unit 31 checks whether or not all of the pixels in the area as set at step S1 have been selected out as being the center pixel If it is verified that not all of the pixels have been selected out as being the center pixel, the program reverts to step S2 to take out another pixel as being the center pixel to prosecute the following processing. If conversely all of the pixels have been taken out as the center pixel, the program moves to step S12.
At step S12, the tap array decision unit 31 detects the coordinates (a, b) voted for the largest number of times at step S10. That is, the straight line passing through the pixel on the frame Fn−1 specified by the coordinates (a, b) and the subject pixel on the frame Fn is detected to detect the straight line segment as the motion vector. The tap array decision unit 31 sends the position information of pixels of the frames Fn−2 to Fn+2, lying on the so detected line segment, to the area extraction units 24, 27.
This terminates the processing.
Since the preset straight line passing through the five consecutive frames is detected as a motion vector, it is unnecessary to carry out the voluminous calculations as needed in the case of block matching.
FIG. 9 shows another illustrative structure of the conventional picture processing apparatus. In the picture processing apparatus 50, shown in FIG. 9, the noisy input picture is sent to each of a filter for reducing the noise of the noisy still picture portion 51, a filter for reducing the noise of the noisy moving picture portion 52 and to a motion detection unit 53.
The filter for reducing the noise of the noisy still picture portion 51 is made up of the components from the amplifier 14 to the frame memory 17 of the picture processing apparatus 10 of FIG. 1, and performs weight addition on the input picture and a picture delayed one frame from the input picture. Thus, if the input picture is the still picture portion, the noise contained in the still picture portion may be reduced.
The filter for reducing the noise of the noisy moving picture portion 52 is made up by the picture processing apparatus 20 shown in FIG. 2 or by the picture processing apparatus 30 shown in FIG. 5, and executes classification adaptive processing based on taps corresponding to the motion vector. So, if the input picture is the moving picture portion, the noise contained in the still picture portion may be reduced.
The motion detection unit 53 detects the motion from the input picture, on the pixel basis, and outputs the results of detection to an output switching unit 54.
If the result of detection from the motion detection unit 53 indicates that the input picture is the still picture portion, the output switching unit 54 selects an output of the filter for reducing the noise of the noisy still picture portion 51 to route the so selected output to outside. If conversely the result of detection from the motion detection unit 53 indicates that the input picture is the moving picture portion, the output switching unit 54 selects an output of the filter for reducing the noise of the noisy moving picture portion 52 to route the so selected output to outside.
This reduces the noise contained in both the still picture portion and the moving picture portion.
However, if, in the case of the present picture processing apparatus 50, the input picture is the still picture portion or the moving picture portion, the processing by the filter for reducing the noise of the noisy moving picture portion 52 or that by the filter for reducing the noise of the noisy still picture portion 51 becomes redundant, respectively.
Consequently, the conventional picture processing apparatus 50 has a drawback that, as described above, the noise contained in the input picture cannot be reduced effectively.
Although it has been practiced to generate picture signals of the fourfold density from the input picture by e.g., linear interpolation, there is raised a problem that linear interpolation leads to a non-acute waveform of the picture signals to give a blurred picture as the result of the linear processing.
Meanwhile, a picture the edge of the feature area of which is not blurred may be generated by applying, for example, the classification adaptive processing.
Referring to FIGS. 10 to 12, the picture processing of generating a picture, the edge of the feature area of which is not blurred, is explained.
FIG. 10 shows an illustrative structure of a picture processing apparatus adapted for generating a picture the edge of the feature area of which is not blurred. In a picture processing apparatus 60, shown in FIG. 10, the input picture, the edge of the feature area (picture area relatively moving to the remaining picture area) of which is blurred, is sent to the a feature area detecting portion 61 and to a classification adaptive processing unit 62.
The feature area detecting portion 61 detects the feature area contained in the input picture to send the detected feature area to a synthesizing unit 63. Meanwhile, the feature area detecting portion 61 detects the feature area so that the pixel density of the feature area will be equal to or higher than that of the input picture.
The classification adaptive processing unit 62 applies classification adaptive processing for removing the noise, correcting the luminance or generating a high definition picture, to the input picture, and outputs the resulting picture to the synthesizing unit 63.
The synthesizing unit 63 synthesizes the feature area from the feature area detecting portion 61 to a picture from the classification adaptive processing unit 62. By taking out the feature area once and synthesizing the so taken out feature area to the remaining picture portion, it is possible to generate a picture the edge of the feature area of which is not blurred.
FIG. 11 shows an illustrative structure of the feature area detecting portion 61. The input picture is sent to a delay circuit 71 and to a motion vector detection circuit 75. The delay circuit 71 delays the input picture, supplied thereto, by a time needed for processing by circuits from the synthesizing circuit 72 to the phase shifting circuit 76, to route the delayed input picture to the synthesizing circuit 72. This allows the synthesizing circuit 72, as later explained, to synthesize the input picture to the corresponding picture.
The synthesizing circuit 72 synthesizes the input picture, supplied thereto from the delay circuit 71, to the phase-shifted picture stored in a storage memory 73 and which is supplied from the phase shifting circuit 76. The synthesizing circuit 72 also routes the so synthesized picture to the storage memory 73.
The storage memory 73 stores the picture supplied from the synthesizing circuit 72 to generate a storage picture, while routing the picture to the detection circuit 74 and to the phase shifting circuit 76.
FIG. 12A shows typical level distribution of pixel values forming the feature area on the stored picture. Thus, in the present instance, the level distribution of the feature area on the stored picture is the same as that of the input picture, however, the level distribution of pixel values of the pixels making up the picture portion other than the feature area is flattened out, as shown for example in FIG. 12B.
The detection circuit 74 detects the feature area from the stored picture supplied from the storage memory 73 to route the so detected feature area to the synthesizing unit 63. Since the feature area on the stored picture has characteristics as explained with base to FIG. 12, the detection circuit 74 is able to detect the feature area accurately.
The motion vector detection circuit 75 is fed with the input picture and with picture data and display positions of the feature area from the detection circuit 74. The motion vector detection circuit 75 detects the motion vector between the feature area from the detection circuit 74 and the feature area in the input picture to route the results of detection to the phase shifting circuit 76.
The phase shifting circuit 76 phase-shifts the storage picture from the storage memory 73, based on the motion vector from the motion vector detection circuit 75, to send the phase-shifted picture to the synthesizing circuit 72.
This allows to generate the picture the edge of the feature area of which is not blurred. However, in synthesizing the feature area with the other picture portions, special processing needs to be carried out at a boundary portion, thus complicating the processing by the synthesizing unit 63.
The picture processing apparatus 60, configured as described above, suffers the problem that a picture of high picture quality, for example, a picture reduced in noise or a picture the edge of the feature area of which is not blurred, cannot be produced satisfactorily from the input picture.